Papers with fine-tuning large language models(LLMs
HyperAdaLoRA: Accelerating LoRA Rank Allocation During Training via Hypernetworks without Sacrificing Performance (2026.findings-acl)
Copied to clipboard
Hao Zhang, Zhenjia Li, Yifan Gao, Xi Xiao, Heng Zhang, Shuyang Zhang, null Xiaoxincc, Bo Huang, Yuhang Wu, Tianyang Wang, Hao Xu
| Challenge: | Low-Rank Adaptation (LoRA) assumes a uniform rank r for each incremental matrix, not accounting for the varying significance of weight matrices across modules and layers. |
| Approach: | They propose a framework that allows for faster convergence of low-rank adaptive models . they use a hypernetwork to prune the outputs of the hypernetworks to generate parameters . |
| Outcome: | The proposed framework accelerates convergence of AdaLoRA by leveraging a hypernetwork. |